AVVO lawyer ratings are statistically unusual

Eugen G Tarnow  February 9 2016 11:23:16 AM

By Eugen Tarnow, Ph.D.
Avalon Business Systems, Inc.
http://AvalonAnalytics.com


Internet ratings are a fascinating topic (I have covered drug ratings before - see Internet drug ratings: AskAPatient.com & WebMD correlate well but Drugs.com does not).  Yelp has recently been in the news with its ratings.  AVVO rates lawyers in a controversial way and has been taken to court but they won.

I have been interested in AVVO ratings because it is in AVVO's interest to rate lawyers who do not use them artificially low and then increase their ratings when the lawyers "claim their profiles," i.e. start to work with AVVO and perhaps pay for their listings. If this is part of their business model one would find statistical properties of the ratings that deviate from what would be the norm.

This entry shows that there is an unusual distribution of AVVO lawyer ratings.  

Typically, a rating scheme would yield a normal curve - i.e. one peak in the middle and slow decrease from that in both directions.  I did a Google search on AVVO ratings today, and found the following distribution for ratings 6.1 and above (the AVVO ratings go up to 10.  I avoided the ratings below 5 because in my Google search the consumer ratings (5 and below) interfered with the AVVO ratings):



Figure 1.  AVVO ratings of lawyers.

There are very large peaks at 6.5, 6.7 and 8 with hardly any lawyers in-between.  The coveted 10 out of 10 spot has a peak but there are almost no ratings from 8.9 to 9.9.

AVVO claims their ratings represent the quality of lawyers according to the following schema:

10.0 - 9.0 Superb
8.9 - 8.0 Excellent
7.9 - 7.0 Very Good
6.9 - 6.0 Good

If that is true the ratings should obey a normal distribution.  There appears to be many lawyers that are "good" and many that are on the bottom of the "excellent" category but hardly anyone who belong in the "very good" category.  And in the superb category, basically everyone is at the top of the category.  All this makes the AVVO ratings statistically highly unlikely.  Or there is something wrong with my Google search terms.

If the business model is based on giving non-cooperating lawyers low ratings and cooperating lawyers high ratings one would expect that higher peaks would belong to cooperating lawyers while lower peaks would belong to non-cooperating lawyers.

Indeed, the first five entries for lawyers rated 8.0 had pictures and their profiles had evidently been claimed by the lawyers.  In the first five entries for lawyers rated 6.7, 2 lawyers had not claimed their profiles.  In the first five entries for lawyers rated 6.5, however, all the profiles had been claimed.

I decided to email one of the 6.5 rated lawyers to find out why he is rated low and found that the AVVO site immediately suggested other lawyers to write to as well.  I decided to try the same with a 10 rated lawyer and no other lawyers were suggested.  That suggests that lawyers with 10 ratings is getting better AVVO services than lawyers with 6.5 ratings.  That might mean that lawyers with 10 ratings pay AVVO more.

This may be a measure of the number of high quality customer AVVO has.  If so this number is 5800.

My investigation only shows that the rating distribution is likely very unusual, not that there is a direct link between payment and ratings.  

Thus it is possible to rate the AVVO rating on statistical plausibility.  We give them 5 stars out of a possible 50 (note that Google can't figure this out and rated this blog accordingly so we changed the rating to a 5...).
 
Comments Disabled